What To Do When A Test Loses

A losing test is a result, not a failure. It prevented a change that would have cost money, and it tells you something true about your customers. The mistake is not losing. The mistake is shipping the variant anyway, or rerunning it until it wins, or quietly leaving it out of the monthly report.

Losing is the normal outcome

Most tests do not produce a winner. ConversionTeam’s audit of 2,288 tests found 19.1% reached statistical significance per test. Optimizely’s analysis of over 127,000 experiments puts the average win rate near 12%.

If four in five of your tests are inconclusive, that is not a broken programme. That is the programme working at industry base rates. A team that reports a win every month is either testing changes so safe they cannot lose, or calling results early.

The three outcomes, and what each one means

The variant lost significantly. Valuable. You had a belief about your customers and it was wrong in a measurable direction. Write down what you now believe instead, because that belief will generate the next three hypotheses.

The variant won but the money did not follow. The trap. Winners in DRIP’s data produced a median 1.88% conversion lift and a median 2.77% revenue per visitor lift, and those two numbers move independently. A conversion win with a revenue loss is a loss. Report it as one.

Inconclusive. The most common and the least useful. Usually it means the effect was smaller than the test could detect, which is a design problem rather than an idea problem.

The questions to ask before you move on

That last row catches more false losses than any other. Check it first.

What not to do

Do not rerun the same test hoping for a different answer. Run it enough times and randomness will eventually hand you a false positive.

Do not ship the losing variant because someone senior prefers it. The whole point of the exercise was to replace that argument with evidence.

Do not stop reporting losses. An agency or team that only shows winners is showing you a selection, and you are paying for the whole programme.

Turning a loss into the next hypothesis

The useful move is to ask why you were wrong, not whether you were wrong.

If you removed a field from a form and conversion fell, the field was doing work you had not accounted for. Perhaps it qualified buyers, or set an expectation, or created a small commitment. That is a real insight about your customers and it generates several new tests immediately.

There is a version of this that goes too far. Not every loss contains a deep lesson. Sometimes the variant was simply worse, and inventing a customer insight to explain an ordinary result is how teams end up with a theory of their audience assembled from noise. If the mechanism is not obvious, record that it lost and move on.

What this looks like across a programme

At a cannabis DTC brand we worked with, three months of testing produced subscription take rate up 75%, average order value up 25% and conversion up 20%, all while paid spend was scaling. Those headline numbers came out of a sequence that included losses. The losses removed whole categories of idea from the roadmap early, which is why the winners arrived as fast as they did.

The habit worth building

Keep a decision log. One line per test: the hypothesis, the result, what you now believe, and what it rules out.

After a year that document is more valuable than any individual winning test, because it is the only asset in the programme that a competitor cannot buy or copy. It is your customers, specifically, described by evidence.

More on how we run and report programmes at Parah Group, including how losses appear in the monthly review.